1. Field of the Invention
The present invention relates to image processing method and apparatus, particularly to method and apparatus for forming a high-resolution image from a low-resolution image, motion vector detecting method and apparatus for use together with the image processing apparatus, method and apparatus for synthesizing a plurality of images, further to a recording medium readable by a computer which is used in these methods and apparatuses, and the like.
2. Related Background Art
Various methods have been heretofore proposed as methods for converting the resolution from inputted low-resolution information to high-resolution information.
In the conventional proposed resolution converting methods, a high resolution is realized by interpolating pixels to the low-resolution information, and the conversion processing method differs with the type of the object image (e.g., a multivalued image in which each pixel has gradation information, a binary image binarized by a pseudo intermediate gradation, a binary image binarized by a fixed threshold value, a character image, and the like).
As the pixel interpolating method in the conventional resolution converting method, a closest interpolating method of arranging the same pixel value closest to an interpolation point as shown in FIG. 1, a common primary interpolating method of determining the pixel value of an interpolation point E by the following operation in accordance with the distances of four points (four point pixel values are set to A, B, C, D) surrounding the interpolation point as shown in FIG. 2, and the like are generally used.
E=(1xe2x88x92i)(1xe2x88x92j)A+i(1xe2x88x92j)B+j(1xe2x88x92i)C+ijDxe2x80x83xe2x80x83(1)
(when the distance between pixels is set to 1, the interpolation point E has a distance i in a transverse direction and a distance j in a vertical direction from A (ixe2x89xa61, jxe2x89xa61)).
Moreover, as heretofore represented by a sampling theorem, means for converting a sampled discrete signal to a continuous signal comprises passing the signal through an ideal low pass filter which can be represented by SINC function, so that the continuous signal can be reproduced. Moreover, since the operation of SINC function requires much processing time, there is proposed another method which comprises approximating the interpolation function represented by the SINC function, and calculating an interpolated value only by a simple operation of sum of products.
For example, in a known cubic convolution interpolating method, the approximating of the interpolation function can be realized. A method of calculating the interpolated value by the interpolating method will be described with reference to FIG. 3. In the pixel arrangement shown in FIG. 3, P denotes an interpolated point (interpolation point), and P11 to P44 denote pixel values of 16 pixels surrounding the point. Then, the interpolation point is interpolated using a cubic convolution function shown in the following equation. Additionally, in the following equation, x{circumflex over ( )}y represents y power of x.                     P        =                                            [                                                f                  ⁡                                      (                    y1                    )                                                  ⁢                                  f                  ⁡                                      (                    y2                    )                                                  ⁢                                  f                  ⁡                                      (                    y3                    )                                                  ⁢                                  f                  ⁡                                      (                    y4                    )                                                              ]                        ⁢                          xe2x80x83                        [                                                            P11                                                  P12                                                  P13                                                  P14                                                                              P21                                                  P22                                                  P23                                                  P24                                                                              P31                                                  P32                                                  P33                                                  P34                                                                              P41                                                  P42                                                  P43                                                  P44                                                      ]                    ⁢                      xe2x80x83                    [                                                                      f                  ⁡                                      (                    x1                    )                                                                                                                        f                  ⁡                                      (                    x2                    )                                                                                                                        f                  ⁡                                      (                    x3                    )                                                                                                                        f                  ⁡                                      (                    x4                    )                                                                                ]                                    (        2        )                                          f          ⁡                      (            t            )                          =                              sin            ⁢                          xe2x80x83                        ⁢                                          (                                  π                  ⁢                                      xe2x80x83                                    ⁢                  t                                )                            /                              (                                  π                  ⁢                                      xe2x80x83                                    ⁢                  t                                )                                              ≅                      [                                                                                1                    -                                          2                      ⁢                                                                        "LeftBracketingBar"                          t                          "RightBracketingBar"                                                ^                                            ⁢                      2                                        +                                                                                            "LeftBracketingBar"                          t                          "RightBracketingBar"                                                ^                                            ⁢                      3                                                                                                            (                                          0                      ≤                                              "LeftBracketingBar"                        t                        "RightBracketingBar"                                             less than                       1                                        )                                                                                                                    4                    -                                          8                      ⁢                                              "LeftBracketingBar"                        t                        "RightBracketingBar"                                                              +                                          5                      ⁢                                                                        "LeftBracketingBar"                          t                          "RightBracketingBar"                                                ^                                            ⁢                      2                                        -                                                                                            "LeftBracketingBar"                          t                          "RightBracketingBar"                                                ^                                            ⁢                      3                                                                                                            (                                          1                      ≤                                              "LeftBracketingBar"                        t                        "RightBracketingBar"                                             less than                       2                                        )                                                                                                0                                                                      (                                          2                      ≤                                              "LeftBracketingBar"                        t                        "RightBracketingBar"                                                              )                                                                        ]                                              (        3        )            xe2x80x83x1=1+(uxe2x88x92[u]) y1=1+(vxe2x88x92[v])
x2=(uxe2x88x92[u]) y2=(vxe2x88x92[v])
x3=1xe2x88x92(uxe2x88x92[u]) y3=1xe2x88x92(vxe2x88x92[v])
x4=2xe2x88x92(uxe2x88x92[u]) y4=2xe2x88x92(vxe2x88x92[v])xe2x80x83xe2x80x83(4)
(In the equation, [ ] denotes Gauss"" notation, and takes an integer portion.)
However, as a result of resolution conversion by the above-described three types of interpolating methods, a blur by interpolation, and a block-shaped jaggy dependent on input low-resolution image occur, and high quality and resolution information cannot be prepared.
To prepare the high-resolution information from the low-resolution information in such background, there is also proposed an interpolating method including a technique of realizing the resolution conversion without generating the interpolation blur attributed to the interpolating processing or the jaggy, a technique of preparing an excellent edge while maintaining the continuity of pixel values, and the like.
However, the resolution conversion by the above-described conventional interpolating method has the following defect. Specifically, even if the high-resolution information is prepared, the enhancement of image quality is limited.
As apparent from the sampling theorem, since the information with the input resolution equaling or exceeding Nyquist limit does not exist in the input image, the preparation of information with Nyquist frequency or more frequency is all based on presumption. Therefore, it is easy to convert flat artificial images such as not-complicated CG image, illustration image, and animation image to jaggy-less images, but it is difficult to enhance the image quality of a natural image by presuming the information equaling or exceeding the Nyquist limit. Specifically, even if any method is used, the image quality of the image obtained by inputting low-resolution information and converting the resolution to a high resolution is evidently deteriorated as compared with the image inputted originally as the high-resolution information.
On the other hand, with the spread of digital video cameras in recent years, it becomes easy to input the picked-up motion image into a computer in the unit of continuous one frame. Therefore, one frame of motion image can also be outputted via a printer. However, as compared with the yearly increasing output resolution of the printer, the input resolution of a picking up system tends to increase, but it is still low in the present situation.
Therefore, as described above in the conventional example, instead of preparing one frame of high-resolution still image from one frame of low-resolution still image, it is considered that one frame of high-resolution still image is prepared from a plurality of continuous low-resolution still images taken from the motion image.
The technique of preparing the high-resolution still image from the low-resolution motion image is proposed in Japanese Patent Application Laid-Open No. 05-260264. The proposed method comprises comparing images continuous in point of time, detecting parameters of affine transformation and parallel movement based on the difference of the images, and synthesizing these images. Additionally, an example in which the synthesizing method is utilized for interpolation is also mentioned.
However, this proposal has the following problem:
Specifically, in the method of utilizing the synthesizing method for the interpolation, by comparing the continuous images enlarged by the interpolating method shown in FIGS. 1 to 3, the parameters are calculated to determine an interpolation position, before performing the synthesis. However, for the enlarged image obtained by the interpolation in this manner, new high-resolution information is not prepared by the interpolating operation itself. Therefore, even when the synthesis processing is performed using the enlarged image in this manner, a really high-resolution image is not necessarily obtained.
Here, the interpolation indicates the interpolation between the pixels. In the interpolation by the synthesizing method, however, when the continuous images are compared, there is not information between the pixels for creating the resolution higher than the resolution of the input pixel. In other words, assuming that two types of images A and B are synthesized, it is difficult to determine the position between the pixels of the image A in which the pixel of the image B is to be interpolated, only by comparing the enlarged images.
This is because the minimum unit of the motion vector amount corresponds to a pixel unit, and there is no resolution finer than a distance between pixels. Specifically, if the vector resolution does not have a precision equal to or less than the distance between the pixels, the effect resulting from the interpolation using a plurality of still images is diminished, and the image quality is substantially unchanged as compared with when one frame of high-resolution still image is prepared from one frame of low-resolution still image as described above in the conventional example.
An object of the present invention is to solve the above-described problems.
Another object of the present invention is to provide an image processing method in which one high-resolution image can be obtained from a plurality of low-resolution images.
To achieve the objects, according to one aspect of the present invention, there is provided an image processing method comprising steps of inputting a plurality of mutually different images having predetermined resolutions, detecting relative positions among the plurality of images with a resolution less than a pixel pitch in the predetermined resolutions, and forming a new image having a high resolution as compared with the predetermined resolutions using the plurality of images in accordance with information indicating the relative positions obtained in the detecting step.
Further object of the present invention is to provide a method in which a motion vector among a plurality of frames can be detected with a higher resolution.
To achieve the object, according to another aspect of the present invention, there is provided an image processing method comprising steps of extracting a plurality of frames from motion images having predetermined resolutions, calculating orthogonal transform coefficients of each of images in the plurality of frames, and detecting motion vectors among the plurality of frames with a resolution less than a pixel pitch in the predetermined resolutions by using the orthogonal transform coefficients.
Still another object of the present invention is to provide an image processing method in which by synthesizing a plurality of images, an image having a high resolution and having no disorder can be formed.
To achieve this object, according to further aspect of the present invention, there is provided an image processing method comprising steps of inputting a plurality of mutually different images having predetermined resolutions, calculating orthogonal transform coefficients of each of the plurality of images, and shifting and synthesizing the plurality of images with a resolution less than a pixel pitch in the predetermined resolutions by using the orthogonal transform coefficients.
Moreover, according to another characteristic of the present invention, there is provided an apparatus or a device which uses the above-described image processing method.
Furthermore, according to another characteristic of the present invention, there are provided a program for realizing the above-described image processing method and a recording medium readable by a computer which stores the program.
The objects and characteristics of the present invention other than the above-described will be apparent from the following detailed description of the mode of the present invention.